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This study explores the relationship between the student performance and 
instructional design. The research was conducted at the E-Learning School at a 
university in Turkey. A list of design factors that had potential influence on 
student success was created through a review of the literature and interviews with 
relevant experts. From this, the five most import design factors were chosen. The 
experts scored 25 university courses on the extent to which they demonstrated the 
chosen design factors. Multiple-regression and supervised artificial neural 
network (ANN) models were used to examine the relationship between student 
grade point averages and the scores on the five design factors. The results 
indicated that there is no statistical difference between the two models. Both 
models identified the use of examples and applications as the most influential 
factor. The ANN model provided more information and was used to predict the 
course-specific factor values required for a desired level of success. 
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Introduction 

For the past few decades, we have witnessed the widespread application of technolog- 
ical developments in the educational systems. The integration of education techno- 
logies with communication technologies introduced e-leaming as an alternative or 
complementary approach for the traditional education methods that are subject to 
time and location constraints. The necessity of having easier access to a wider range 
of information, coupled with the demand for effective and low cost education, paved 
the way for the increased use of the Internet in learning environments. One of the 
challenging problems that society has been facing is the need to improve the quality 
of e-learning systems. 

A number of studies have been conducted to gain a better understanding of the 
factors that affect the success of e-leaming systems and the interrelations of these 
factors. The literature can be categorised into three main groups, namely, student and 
instructor characteristics (Bozarth, Chapman, and LaMonica 2004; Greene et al. 2004; 
Hillsheim 1998; Kerr, Rynearson, and Kerr 2006; Liaw, Huang, and Chen 2007; 
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Marom et al. 2003; Soong et al. 2001), instructional design factors (Bozarth, 
Chapman, and LaMonica 2004; Wegner, Holloway, and Garton 1999), and demo- 
graphic variables (Diaz 1999; Diaz 2000; Dille and Mezack 1991; Hillsheim 1998; 
Lynn 2002; Marom et al. 2003; Muse 2003). 

Of these success factors, good instructional design is seen as a necessity for the 
establishment of efficient learning environments. Instructional design factors can be 
defined as the optimum design of the instructional environment to meet the require- 
ments of both the students and the instructors (Qureshi 2004). Instructional design, 
from the Web-based learning perspective, is a three-dimensional process involving 
designers, students, and instructors (Wegner, Holloway, and Garton 1999). The 
designers provide the students with the appropriate course content through working 
collaboratively with the instructors. Fresen and Boyd (2005) conducted a study 
regarding the critical success factors of Web-based learning systems and they synthe- 
sised a list of success categories and associated factors from several e-leaming 
resources. For the category of instructional design, they identified 15 factors, which it 
is possible to divide into two groups: content design-related factors and other instruc- 
tional design factors (Table 1). The first group of factors is more about the interface 
design aspect of e-leaming contents while the latter group is associated with the 
usability of content and student interaction. 

The literature already offers a range of recommendations for the first group of 
interface design factors, also known as content development standards, concerning 
issues such as the design of Web interface (Rollins 2002), texts (Ardac and Unal 
2008; Bullock, Geraci, and Bear 2003; Ipek2001; Lee 1999; Qureshi 2004; Pomales- 
Garcia and Liu, 2006) multimedia applications such as graphics (Angeli and 
Valanides, 2006; Jennings 2001; Qureshi 2004; Swan 2001), animations, video files, 
and colours (Angeli and Valanides 2006; Bhowmick et al. 2007; Bullock, Geraci, and 
Bear 2003; Ipek 2001; Jones 2002; Paul 2001; Qureshi 2004), practical exercises and 
illustrative examples (Jennings 2001; Paul 2001). We already know that animations 
should be designed to be interactive, complementary, and supportive to the text docu- 
ments (Bullock, Geraci, and Bear 2003; Ipek 2001), but that the excessive usage of 
graphics and animations may distract the attention of the students from the course 
(Ipek 2001). This study will assess the relative importance of such factors on student 
performance. 

Table 1 . Instructional design factors. 


Factors related to content design Other instructional design factors 


• Rich learning resources/sound learning 

materials 

• Design standards/guidelines/minimum 

requirements 

• Manageable segments/modular/chunking 

• Appropriate use of images, graphics 

• Appropriate layout and presentation 

• Appropriate bandwidth and download 

demands/speed 

• Purposeful use of learning media 

• Routine review and evaluation of 

courses/products 

• Offer a complete learning package 


• Usability/minimise student frustration/ 

appealing 

• Co-operative/group learning/team work/ 

reciprocity 

• Student engagement in higher cognitive 

levels/knowledge construction/ challenges 

• Enhanced student motivation/ 

responsibility for own learning 

• Inclusivity: social, cultural, gender, 

disabilities 

• Interactivity/active learning/learning 

activities 


Source: Fresen and Boyd 2005. 
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Methodology 

The success of a learning system depends on the level of efficiency in which students 
can grasp the contents of their courses. Since measuring this efficiency may not be 
easy to formulate, grade points are seen as indicators for students’ success. The aim 
of this study is to establish a relationship between instructional design factors and the 
student success level. This relationship is investigated by making use of statistical and 
neural networks techniques. This section will describe issues regarding the variable 
selection, data collection and sampling, and data analysis. 


Creation of criteria 

Having considered the instructional design factors particularly related to content 
design from the relevant literature, a list was created of 1 3 variables that are thought 
to be have an effect on student success. These variables were emailed to 20 inter- 
viewees a week ahead of a semi-structured individual interview. Interviewees were 
experts who had been working on the project for more than five years. Eighty percent 
of the participants had an academic background in instructional design while the rest 
mostly had a design-oriented background. The email included an invitation for the 
semi-structured interview together with information about what they were asked to 
carry out before attending the interview. A text file including some explanation of the 
variables and their brief description was also attached. The main aim of the email 
communication was to ask the interviewees to consider the suitability of those 
variables as instructional design criteria. 

During the interview, the interviewees were also allowed to combine or eliminate 
some of the variables to create a reduced set. All possible instructional design factors 
suitable for the current project were extracted. The decision criterion for selecting 
those factors was based on the major agreement of the participants for each criterion. 
Of these 13 variables, two were eliminated by all the participants and the rest were 
reduced to five factors. 

At the end of the interview stage, the following criteria were selected as predictors 
of success: the length of the texts, the usage of examples and applications, exercises, 
quiz and homework, multimedia applications, and visual integrity. These criteria were 
utilised by the select models as independent variables, while the average grade of 
students in different courses was used as a proxy for student success. However, in 
the second implication of the artificial neural network (ANN) model, a different 
operationalisation was employed, which will be presented later. 


Sample 

The data used to perform the analyses were collected from the e-leaming school of a 
university in Turkey. The school consists of five different study programmes with 
more than 100 courses, serving more than 1000 students in total. Only one study 
programme is business-related while the rest of them are technically oriented. 

A sample of five courses was selected from each programme for the data analy- 
ses, giving a total of 25 courses. Each of the five courses is programme-specific and 
considered as representative for the corresponding programmes. The sampling 
process ensured that the courses chosen have different types of content covering a 
wide range of subjects and the sample includes courses with a variety of difficulty 
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level for each specific programme. More than 100 students were enrolled in each 
course. 


Data collection 

The data collection had two parts. The first was conducted to obtain the instructional 
design scores. The courses were evaluated by the experts on each of the selected 
instructional design factors. Second, students’ grade point averages were collected 
for the corresponding courses directly from the student registration office of the 
university. 

The first part of the data was collected from the experts in the form of a survey. 
The survey was used to obtain the scores on the five instructional design criteria, for 
each of the 25 courses. The experts were asked to evaluate each course based on the 
selected criteria by giving points on a scale between 0-10. The average grade was 
based on overall 100 points. The results of these evaluations as an average of all 
opinions (for five random courses) can be seen in Table 2. In Table 2, the first column 
shows the codes which were developed for the puipose of differentiating the courses. 

The length of the text was compared to already determined standards. The related 
literature suggests that the optimal length of the texts should be 8-10 pages, given that 
anything between six and 12 pages is acceptable (Bullock, Geraci, and Bear 2003; 
Ipek 2001). The length of the texts was categorised into three classes and for each of 
them a ranking point was assigned. Conversion of the total average rankings into the 
10-point scale provided the final results for this variable. 

Examples and applications, and exercises and homework were rated by dividing 
the number of weeks where these factors were demonstrated by the total number of 
weeks in the course. Regarding the usage of multimedia applications, this factor was 
assessed based on the usage of graphics and animation. Two usage categories were 
considered and the rankings corresponding to that categorisation converted to an 
average usage of both animation and graphics, which was taken as a final point value. 

The experts evaluated visual integrity intuitively as an average of the evaluations 
given to the quality of colour competence, the inclusion of multimedia applications 
and the texts in the whole content, and the errors occurred in the graphics and 
animations used in the texts. 

The final column represents the weighted average of the final marks of homework, 
quiz, and final examination for a given course. The percentage of final examination is 


Table 2. Scores for each course for the five instructional design factors. 


Course 

code 


Pointing results 



Average 

grade 

Length 
of the 
text 

The usage of 
examples and 
application 

Exercises 

and 

homework 

Multimedia 

applications 

Visual 

integrity 

30-105 

6.4 

5.0 

9.3 

5.7 

7.0 

49.57 

32-101 

4.3 

5.7 

0.7 

8.6 

6.0 

45.54 

34-101 

5.0 

5.7 

2.9 

6.8 

6.0 

41.38 

36-101 

6.0 

6.4 

1.4 

7.9 

7.7 

58.60 

38-105 

6.0 

5.0 

0.7 

6.8 

6.7 

45.38 
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80% while homework and quiz marks take 20% overall. The study assumes that the 
questions that were asked in the quiz and final exam were extrapolated from the 
course content. 


Data analysis 

The relationship between the criteria and the success of students was analysed using 
two different prediction methods, a multiple-regression analysis technique and an 
ANN model (multi-layer perception). These prediction methods have already been 
used in several comparative studies in different areas (Baker and Richards 1999; 
Gaudart, Giusiano, and Huiart 2004; Guiterez-Estrada et al. 2004; Hardgrave, Wilson, 
and Walstrom 1994; Hyvarinen and Bingham 2003). Recent studies have considered 
ANN as an alternative method of traditional data analysis techniques or perhaps 
a more-developed version of these techniques. Although the solution of an ANN 
may not be the optimum one, its results can be considered rather satisfactory as 
compared with other traditional methods (Hertz, Krogh, and Palmer 1994; Oztemel 
2003). 

The ANN prediction models used in the current study are presented below. The 
models’ network typology consists of three layers, namely input, hidden, and output 
layers. In the first model, the input and the hidden layers have five nodes each while 
the output has one node only (see Figure 1). The second model’s typology consists of 
one input node, one hidden node, and five output nodes (see Figure 2). 


Numerical illustrations 

This section provides some illustrations with regards to the implementation of the two 
prediction models on the dataset as well as the associated results obtained from the 
analyses. 



Input H.iJden Oi.Jl.pi, ! 

layer Layer Uyer 


Figure 1. Typology of feed-forward neural network model. 
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Figure 2. Typology of back- forward neural network model. 


Multiple-regression model 

After employing the regression analysis, the coefficient of determination (R ) was 
found to be 0.8067 and it indicated that the five criteria used in the model can explain 
80.67% of average grade points of all courses. The resulting F-value, which was 
15.859 with a probability of 0.00003, leads us to reject the null hypothesis (H 0: The 
regression coefficients of the model are all equal to each other that have a value of 
zero). Furthermore, a t-test analysis was perfonned within a 95% level of confidence 
to check whether the regression coefficients were significant or not. The results of the 
i-test can be seen in Table 3. In the table, X ls X 2 , X 3 , X 4 , and X 5 were used to refer 
to the length of the text, the usage of examples and applications, exercises and home- 
work, multimedia applications, visual integrity respectively. 

As it can be seen from the Table 3, all coefficients are statistically significant. 
Therefore, the regression model can be as follows: 

y = 2.9125x 1 + 3.8187x 2 +0.4530x 3 + 1.4641x 4 + 3.1639x 5 21.6215 (1) 

The expected and the obtained values were compared using a z-test, and it was 
concluded that there is no statistical difference between them (t = -0.006, p = 0.99). 


Table 3. Regression model coefficients. 


Variable 

Coefficient 

Z-value 

p-value 

XI 

2.912 

3.067 

0.006 

X2 

3.819 

3.201 

0.005 

X3 

0.453 

1.949 

0.048 

X4 

1.464 

2.251 

0.036 

X5 

3.164 

2.126 

0.047 

Constant 

21.622 

-2.280 

0.034 
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The results of the regression analysis indicated that the five instructional design 
factors are related to students’ success. It can also be concluded that the most effective 
factor is the usage of examples and applications. 


Multi-layer perceptron models 

The ANN prediction model was designed in two different ways. It can be applied in a 
way similar to that of the multiple-regression model, which considers the instructional 
design factors as input and then tries to predict average grade points. It can also be 
applied inversely, where the average grade points are used as an input and therefore 
ending with the instructional design factors as an output. In the following sections, the 
implementation of these models on the selected data will be demonstrated. 


Feed-forward prediction 

The first ANN model predicts the average grade points via the multi-layer perceptron 
method. Hence, the dataset was scaled within the interval of 0-1 . The dataset was 
divided into three sub-sets, the training set, the test set, and validation set. Accordingly, 
the data collected from the 25 samples were divided into 10 training, five test, and 10 
validation sub-sets. For homogeneity purposes, the dataset was divided over the 
courses in the three divisions. Given the unavailability of a large sample, the error 
value was selected as high as 0.07. Table 4 shows the prediction input sets, the 
expected outputs, the predicted outputs and the error percentage, which represents the 
difference between the prediction results and the expected values. 

After a series of simulation studies, the network reached 100% learning and 100% 
test success in the 100000th iteration with a learning coefficient of 35% and a momen- 
tum coefficient of 75%. In order to evaluate whether there was a significant difference 
between the expected and the predicted values, a t-test analysis was performed. The 
t-test value (t = 0.04, p = 0.96) revealed statistically significant results. 

It is possible to see from Table 4 that the predicted values are close to their corre- 
sponding expected values. Only for the second and the fourth cases, the error values 


Table 4. The differences between the expected and predicted results for the first multi-layer 
perceptron model (five inputs, one output). 

Prediction dataset 


Input 1 

Input 2 

Input 3 

Input 4 

Input 5 

Expected 

output 

Predicted 

output 

Error (%) 

0.057 

0.057 

0.000 

0.075 

0.070 

0.494 

0.505 

0.009 (1.8%) 

0.071 

0.085 

0.000 

0.089 

0.083 

0.687 

0.741 

0.054 (7.8%) 

0.040 

0.057 

0.100 

0.068 

0.073 

0.493 

0.479 

0.014 (2.8%) 

0.043 

0.057 

0.007 

0.086 

0.060 

0.455 

0.431 

0.024 (5.2%) 

0.033 

0.061 

0.100 

0.068 

0.073 

0.495 

0.478 

0.017 (3.4%) 

0.064 

0.057 

0.000 

0.073 

0.077 

0.563 

0.554 

0.009 (1.5%) 

0.050 

0.071 

0.000 

0.088 

0.077 

0.581 

0.595 

0.014 (2.4%) 

0.050 

0.057 

0.086 

0.057 

0.063 

0.461 

0.458 

0.003 (0.6%) 

0.067 

0.050 

0.000 

0.079 

0.063 

0.491 

0.480 

0.011 (2.2%) 

0.067 

0.057 

0.000 

0.043 

0.067 

0.494 

0.490 

0.004 (0.8%) 
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are slightly high. A reasonable explanation for this could be that the sample size for 
training was not large enough. Even though sample size for training was not sufficient, 
the worst prediction result of the network had 7.8% of error value. Hence, it can be 
said that multi-layer perceptron model was performed successfully. 


Back-forward prediction 

In this part of the study, the second network model uses the first model’s inputs as 
output and ends up with the inputs of the previous model. When the previous output 
values were used as input, the network was able to produce the instructional design 
factors’ values for a desired interval value of an average grade point. 

Similar to the first network model, a 0.07 error value was selected. The network 
trained had a 100% of learning success and 35% and 75% of learning and momentum 
coefficients respectively. With these parameters, the network produced the results as 
shown in Table 5. 

In general, the model produced very close results to the expected values, except 
for output 3. The reason for the high difference between the expected and the network 
output is that output 3 had a wide range of value varieties, which caused a learning 
problem in the network and an explanation difficulty. 

The back- forward prediction model provides insights about the type of instruments 
that should be included in the courses. For a desired success level, a reasonable combi- 
nation of the instructional design factors levels can be predicted. 


Discussion 

This study investigated the effects of instructional design factors on students’ success 
using multiple-regression and ANN approaches as prediction models. These models 
were developed and compared using the same prediction dataset. The results indicated 
that all of the five factors have positive effects on students’ success. Additionally, the 


Table 5. The differences between the expected and predicted results for the second multi- 
layer perceptron model (one input, five outputs). 


Input 

Output 1 

Output 2 

Output 3 

Output 4 

Output 5 

EO 

NNO 

EO 

NNO 

EO 

NNO 

EO 

NNO 

EO 

NNO 

0.494 

0.057 

0.053 

0.057 

0.058 

0.000 

0.031 

0.075 

0.068 

0.070 

0.067 

0.687 

0.071 

0.068 

0.085 

0.073 

0.000 

0.047 

0.089 

0.074 

0.083 

0.082 

0.493 

0.040 

0.053 

0.057 

0.058 

0.100 

0.031 

0.068 

0.068 

0.073 

0.066 

0.455 

0.043 

0.043 

0.057 

0.056 

0.007 

0.028 

0.086 

0.067 

0.060 

0.064 

0.495 

0.033 

0.033 

0.061 

0.058 

0.100 

0.031 

0.068 

0.068 

0.073 

0.069 

0.563 

0.064 

0.058 

0.057 

0.063 

0.000 

0.036 

0.073 

0.070 

0.077 

0.072 

0.581 

0.050 

0.059 

0.071 

0.065 

0.000 

0.037 

0.088 

0.071 

0.077 

0.073 

0.461 

0.050 

0.051 

0.057 

0.056 

0.086 

0.029 

0.057 

0.067 

0.063 

0.064 

0.491 

0.067 

0.053 

0.050 

0.058 

0.000 

0.031 

0.079 

0.068 

0.063 

0.067 

0.494 

0.067 

0.053 

0.057 

0.058 

0.000 

0.031 

0.043 

0.068 

0.067 

0.069 


Note: EO, expected output values pertaining to the input values of the second model; NNO, predicted 
values produced by the network. 
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multimedia applications and the visual integrity factors have strong relationship with 
the success of the students, and the usage of examples and applications is the most 
effective among all factors. By means of the multiple-regression and the feed-forward 
multi-layer perceptron model, a decision concerning the predicted level of success of 
a course can be made before displaying the course online. For example, necessary 
improvements for a course can be taken into consideration during the pre-displaying 
process. These models can be used as a strategic tool to decide whether to present a 
course online or not as well as which aspects of it should be improved. 

Table 6 summarises the results of both models and the comparison between them, 
along with their expected outputs based on the error values for the dataset. 

A paired /-test analysis was conducted to evaluate whether the results of 
two models were different from each other or not. According to the /-test results, 
(t = 0.696, p = 0.504), there was no statistical difference between the two models. 

However, it should be noted that the results of the second ANN model (back- 
forward) cannot be possible to be produced by the multiple-regression model. In this 
manner, ANN models can be chosen instead of a multiple-regression approach due to 
their capability of giving different aspects of this kind of problem. The back-forward 
neural network prediction model stimulates us to look at the answers for the question 
of what design structure a course content should have to make the learning outcomes 
adequate for the students to lead them to success. 

The contribution of this study is two-fold. By introducing the application of ANN 
to the design of learning environments, it might be possible for the researchers to 
introduce new problem domains which utilise the capability of this technology. With 
regards to the contribution of the study for practitioners, the study brings out the impli- 
cations for an ongoing e-leaming system. To be able to reach the highest success level 
on student performance, several simulation studies can be performed by investigating 
the effects of different success predictors, not just instructional design factors, on 
students’ success. The simulation could be performed based on the feed-forward 
neural network model. Furthermore, the simulation can be expanded through the 
usage of back-forward network model by including the attributes of the learners 
together with the other success measurements. As a result of this, an appropriate 
design structure can be tailored for each group of learners. 


Table 6. Comparison of regression and ANN models. 


Expected output 

ANN approach (feed-forward) 

Regression approach 

Predicted results 

En-or (%) 

Predicted results 

Error (%) 

49.4 

50.5 

0.9 (1.8%) 

49.9 

0.5 (1.0%) 

68.7 

74.1 

5.4 (7.8%) 

71.1 

2.4 (3.4%) 

49.3 

47.9 

1.4 (2.8%) 

49.5 

0.2 (0.4%) 

45.5 

43.1 

2.4 (5.2%) 

44.5 

1.0 (2.1%) 

49.5 

47.8 

1.7 (3.4%) 

49.0 

0.5 (1.0%) 

56.3 

55.4 

0.9 (1.5%) 

53.8 

2.5 (4.4%) 

58.1 

59.5 

1.4 (2.4%) 

57.2 

0.9 (1.5%) 

46.1 

45.8 

0.3 (0.6%) 

47.0 

0.9 (1.9%) 

49.1 

48.0 

1.1 (2.2%) 

48.5 

0.6 (1.2%) 

49.4 

49.0 

0.4 (0.8%) 

46.9 

2.5 (5.0%) 


30 H.I. Cebeci et al. 


Conclusion 

Even though the success of an e-leaming system is explained by factors such as the 
number of students and lecturers in the system, their learning and teaching efficien- 
cies, and education technologies used, students’ success level is another significant 
indicator. Students’ success level is affected by socio-economic, demographic and 
instructional design factors. In this study, only instructional design factors were inves- 
tigated. As a result of the study, it is recommended for instructional designers and 
lecturers to prepare course contents within a specified standardisation and to make 
efficient use of multimedia applications. Furthermore, visual design of a course should 
always be taken into consideration. 

During this research, uncontrollable variables such as socio-economic and demo- 
graphic factors, which are indirectly related to students’ success, have not been 
considered. This remains a limitation of the study. For future work, a statistical anal- 
ysis of how these factors affect the students’ success can be investigated along with 
the investigation that has been done in this study. Furthermore, as an indicator of 
success, not only grade point averages but also some other success measurements after 
graduation should also be included in further studies. 
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